Repository navigation
fix(executorch): move compile_and_save's engines into the program instead of copying them - #5045
Draft
shoumikhin wants to merge 1 commit into
Draft
shoumikhin wants to merge 1 commit into
shoumikhin wants to merge 1 commit into
Conversation
shoumikhin
force-pushed
the
compile-and-save-consume
branch
from
November 15, 2026 01:40
03b81ce to
ba01aca
Compare
shoumikhin
force-pushed
the
compile-and-save-consume
branch
2 times, most recently
from
November 15, 2026 07:04
3cf1702 to
306a9de
Compare
shoumikhin
marked this pull request as draft
November 15, 2026 07:04
shoumikhin
force-pushed
the
compile-and-save-consume
branch
8 times, most recently
from
November 19, 2026 11:23
2f360b9 to
d53ffa2
Compare
…tead of copying them `compile_and_save` writes each engine to a temporary file as soon as it is built. The ExecuTorch save then copies that file into the `.pte` (or the `.ptd`), and the file is deleted only when the call returns. So while the program is written, every engine is on disk twice. For a model whose engine is tens of gigabytes, the copy needs that much more free disk, and an export can fail on a device that has room for the result. The backend now hands each of these engine files to ExecuTorch as data that owns the file (`FileBackedData(path, cleanup=True)`); it is deleted after the save anyway. Inside `compile_and_save`, when the outputs would not fit in the free space of their file system (with a tenth to spare, and an output already there counted as free, since both a copy and a move replace it), the program is then written with ExecuTorch's `save(file_path, consume=True)` and the `.ptd` with `write_tensor_data_to_file(..., consume=True)`, so ExecuTorch moves each engine into the program: it copies it back to front and truncates the file as it goes, so the engines are on disk once (plus one chunk) instead of twice, on a file system with sparse files. ExecuTorch moves other data that owns its file the same way, such as the CUDA backend's weights, and copies data it can't move, with a warning. Its `save` takes only a `.pte` path, so a program saved under another name is written as before, with a copy; the `.ptd` still moves. ExecuTorch is an optional extra, and releases before pytorch/executorch#25099, such as 1.6.0, have no `consume` argument. Both writes check whether ExecuTorch's method takes it, through one small helper, and copy as before when it doesn't. That fallback writes the program through `open` and `write_to_file`, as before this change, so a failed write still raises. The move is slower than a copy, because ExecuTorch syncs each chunk to disk before the engine file gives it up. Measured on btrfs, engines of 0.5, 2 and 8 GiB took 1.3, 8 and 37 seconds to move and 0.5, 1.9 and 6.7 seconds to copy. So `compile_and_save` moves only when a copy would not fit, where the save failed before, and otherwise copies exactly as before. The docstring says so. If a move fails after some data left its engine file, ExecuTorch keeps its partial output, `<name>.<random>.partial`, which holds that data. The engine files are temporary and are deleted when `compile_and_save` returns, so that output can never be finished, and it can be as large as the engines on a disk that just ran out. So the consuming writes now run inside `moving_engines_into`: when the save fails, it deletes each partial output this save left, and a note on the error says to export again. A partial output that was there before the save is never touched. ExecuTorch can also leave other data it moved in part, such as a file of CUDA weights whose rest is in the partial output; that file is deleted too, since nothing can finish it either. Afterwards it empties the plan of each engine the save moved, in part or in full: that plan maps an emptied file, so a later read of it through its module, such as a second save in the block, killed the process with a bus error. Now the plan is empty and that read raises. An engine file the cleanup can't delete is logged as a warning, so it can't hide the error that ended the block, and a save that succeeded doesn't fail over it. The program path goes through `os.fspath` first, so a `pathlib.Path` still works. With `generate_etrecord`, the save copies the engines instead. The ETRecord is written after the program and reads the engines, so a move would leave it nothing to read. The program is still written first, as before. The docs now say why `compile_and_save` can write an ETRecord when a plain save of a program with TensorRT engines can't: it keeps each engine as a byte buffer read from its file. The record then holds copies of every engine, and writing it reads them into memory, which the docs also say: for a 4 GiB engine the record took 8 GiB, and the save 3 GiB more host memory. The move needs the `consume` argument of `save` and `write_tensor_data_to_file` from pytorch/executorch#25099; with an ExecuTorch without it, `compile_and_save` copies as before. Only the move passes `consume`, so every other save calls ExecuTorch exactly as before. Test plan: - `tests/py/dynamo/executorch/test_api.py`, CPU, with the lowering stubbed: - New `test_save_in_the_offload_block_moves_the_engines_when_a_copy_does_not_fit`: inside `compile_and_save`'s offload block, the program and the `.ptd` are written with `consume=True` when the disk has no room for a copy, and without it when it has, counting an old output it replaces as free. The path is a `pathlib.Path`. - New `test_save_in_the_offload_block_moves_only_the_ptd_into_a_path_without_pte`: the same block with a `.bin` path writes the program without `consume` and the `.ptd` with it. - New `test_save_in_the_offload_block_copies_with_an_executorch_without_consume`: with an ExecuTorch whose writes take no `consume`, the program goes through `write_to_file` and the `.ptd` is written without `consume`. - New `test_a_failed_move_in_the_offload_block_deletes_its_partial_output`: a consuming `.pte` or `.ptd` write that stops inside an engine leaves no partial file, and the error says to export again. - New `test_a_failed_move_deletes_what_it_moved_in_part`: a failed save deletes the partial output it left and a data file it moved in part, and says to export again; an untouched data file and a partial output from before the save are kept; the engine files are deleted in every case. - New `test_a_save_empties_the_plans_of_the_engines_it_moved`: after a save, a moved engine's plan is empty and no longer found as an engine file, the other one is untouched, and a second save passes over the moved one. - New tests for engine files the cleanup can't check or delete: a warning, not an error. - New `test_save_in_the_offload_block_keeps_the_program_when_the_etrecord_fails` and `test_save_executorch_does_not_consume`. - `tests/py/dynamo/executorch/test_compile_and_save.py` on an A100 with TensorRT 11.4, with ExecuTorch 1.6 and with ExecuTorch from pytorch/executorch#25099: - The streaming test also checks that each engine reaches ExecuTorch as data that owns its file. - New `test_engines_move_into_the_program_without_a_second_copy`, new `test_an_etrecord_makes_the_save_copy_the_engines`, and new `test_a_failed_move_leaves_no_partial_output` (the disk fills after the first engine moved; no partial file is left). They make the disk look full, so the save moves, and they need an ExecuTorch that moves data. The skip checks for `Cord.file_backed_buffers` as well as `consume`, so an ExecuTorch that renames `consume` fails them instead of skipping them. Signed-off-by: Anthony Shoumikhin <anthony@shoumikh.in>
shoumikhin
force-pushed
the
compile-and-save-consume
branch
from
November 20, 2026 13:04
d53ffa2 to
bc9748e
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on pytorch/executorch#25099, and lands after it with the ExecuTorch pin moved, so CI runs the move tests. With an ExecuTorch without
consume, such as 1.6.0,compile_and_savecopies as before. Other saves are unchanged.What is wrong today
compile_and_savewrites each engine to a temporary file as soon as it is built. The ExecuTorch save copies that file into the.pte(or.ptd), and deletes it only at the end. So every engine is on disk twice, and a large export can fail on a device with room for the result.What this change does
FileBackedData(path, cleanup=True)).compile_and_save, when the outputs would not fit in the free space of their file system (with a tenth to spare), the program and the.ptdare written withconsume=True. ExecuTorch then moves each engine into them, so the engines are on disk once, on a file system with sparse files. Other data that owns its file, such as CUDA weights, moves too.<name>.<random>.partial. It can't be finished, so it is now deleted, with any data file the move cut short (such as CUDA weights), and the error says to export again.generate_etrecord, the save copies, because the ETRecord reads the engines afterwards. The docs now say whycompile_and_savecan write an ETRecord (it keeps each engine as bytes read from its file) and that the record holds copies of every engine.Tests
test_api.py, CPU, lowering stubbed: the move only without room for a copy, the fallback, partial outputs deleted after a failed move, emptied plans, and engine file cleanup.test_compile_and_save.py, A100, TensorRT 11.4, with ExecuTorch 1.6 and with Make save raise on failure; let save move file-backed data into the output instead of copying it executorch#25099: with the disk made to look full, engines move with no copy, an ETRecord makes the save copy, and a disk that fills mid-move leaves no partial file.